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ABSTRACT 

The development and validation of oral native 
language tests for Dutch adolescents was conducced for both 
assessment and curriculum development purposes at the secondary 
level. The oral language situations tested were the monologue, 
dialogue, and pclylogue or group conversation on a topic. An American 
inventory of common speech subjects was used to construct test 
situations, and the test subskills were drawn from the students* 
textbooks. The situations, subjects, and subskills were combined into 
three tests, which were administered to 14-year-olds in all areas of 
the country and all dialect regions. The tests were taped, and 
teachers were asked to rate the dialogue and monologue tests on 
content, organization, language, delivery, and communication. The 
interrater reliability and homogeneity of assessment criteria were 
analyzed statistically. On the whole, ic was found that the two test 
types were closely correlated and that overall scores would have 
sufficed for ranking students in the case of almost all raters. 
However, when only one rater is available, as *n a classroom testing 
situation, the analytic method of scoring is preferred to the overall 
scoring method. (MSE) 
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. 1 Introductlor. 

For oral tests to be used In the course of the education process 
and meant to supporf It, that is for formative oral testSr as a rule ' 
• only direct and not Indirect testing methods will be adopted. That Is 

J^H*^:^ L^*^^r?.^l!*^u*? J''!! ^®!^ PwPlls ^'lll actually have to speak. 
And, Indeed, I think that Indirect measurer of the oral skill at an 
Intermediate level should be avoided, not only or In the first place 
for reasons of valliitty but with a view to 
* the motivation or pupils, as well as 
- the effects such tests have on education, whether Intended 
or not. 

^4/^6enepally speaking testconstructors prefer their tests to 
, elicit behaviour that does not differ too much from the crtterlon 
. owvlour. the sort of hehavlour that would occur In ordinary life 

^^^iSTl^rJ** ^" ^ ^ In everydny 

reality there are, generally speaklna, three situations calling 

*^ fK?!Si*J''**^^V^*J***,®'*^ ^^V^' the monologue, the dlalooie 
^ polylogue. In the large majority of situations falling into 
t;,.any of these three main categories, each speaker can. to a large 
^extent, deteralm: the direction of the activity in which he is 
engaged. This Is selfevident In ca^e of the monologue; in the other 
^???* the influence exerted by the participants will not exactly be 
alike, but however that may be. it cannot be predicted with any 

JS^I?^?.^*^ ^ ^P***^*" ^ even situations 

thatllmit a speaker's free scope severely such as Interviews. e,g. 
between a senior and a junior staff member, even such situations remain 

event In which he participates to take an unexpected turn. «f It Is 
thought Important that the behaviour elicited by the test and the 
criterion behaviour do not diverge too much, the test should create 
a situation which leaves the testee the scope he would have In 
everyday reality. 

Apart from this, 'realistic' tests will give rise to fewer 
methodological problems In validating tests, that is. provided the 
test s reliability neets the proper requirements. I take for 
granted that & test which Is reliable and prompts behaviour that Is 
quite close to criterion behaviour can be considered e valid test. 
Only if the behaviour required by a test deviates more markedly 
from the behaviour that Is the real object of the evaluation, 
further validation Is called fbr. 

Of course I should make mention at this point of a differenca 
between testing the oral skills of LI &..d L2 learners. LI learners 
can be ussumed to avail themselves of a certain latitude In giving 
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dirtctlon to i conversitlor. But L2 leimers it in elaMntiry Uvtl 
>rtn not rtgrtt tlit constrtlnt;; put upon their frttdos of 
•xprtsslon: tiMo vould net know to ust It If It wtrt grinttd to thti. 
That Is why itHctly stnicturtd tests of slapU vocibuUry. 
pronunclitlon or stnttnct structure ire not it v^rlince with the 
9enert1 principle of illowlng the testee free scope. IncoMpetlblHty 
only irises it in idvinced level of IZ proficiency. At this level 
requlroMnts will be Mtft of the L2 learner thet com close to those 
Mde of the LI leimer. Uhit I m going to s«y ibout oral tests of Dutch 
for 14 year old Dutch pupils «1ght consequently be of Interest to 
those concerned with testing L2 proficiency. 

Flnt of all I shall say one or two thi NMit the starting 
points adopted fbr the construction of the te. ^ I shall 
discuss the Ust foraatt and present the resuU etests 
and rating tryHMits. 



2 Test wontent 

The oral tests fo>^ native speakers of Dutch developed so far 
are part or a plan to coapose a Mt of ora) tests fbr the whole 
of secondary education* Tests for Interatdlate levels have been 
constructed first, because It Is supposed that a seHes of such 
teeu would oontrlbuU to the developaent of proirwsd Instruction 
In the Orel shllK That Is to sior* Individual teachers as well 
as wHtert of eduutlonal seterlal could benefit fro« the existence 
0/ e eet of Ibmtlve tests based on relevant educational objectives. 
An In^a nto ry of educational obJectlvM through an analysis of aost 
frequently used toftttooks showed a laclt of eiiy sort of systeaetic 



The series of tests was set up as follows, 
i Orel proflclenqr was defined as: the skill to express onnelf 
erally In au adequate MMier In functional language situations. 
The latter are considered to be Imguage situatlom derived frea 
an analjfsis of requlraaanU uede of pupils both et school and 
out of Mhool. The aaln cHteHon fbr cetegorliing these 
sftuatloee Is the nui*er of speekers 'involved: aM0lo«ue, dialogue 

oolyloeue* Cecb of tHe three aaln divisions Is to cowHse 
about too languige slUsetfona fn e dce rde nce with Vm subcategories 
Ming u^ the hoedlufs of ■onetogue, dialogue end polyloguo. 
t In Mloctlog relevenr subjects ood situations wa used the 
Inventory presented In *WMic$ In smoking and listening for High 
School iioduetos* by Renold C. Bessott et a1. In Cooounlcatlon 
EAicatlon (If/St Its f). uhlch Is an outcoae of the alnlaal 
cevetooclos ■oveaent In the USA. It lists all the situetlons 
celllnf fbr orel speech In which tho average (Aaerlcan) citlien 
■ly gei'lnvoluod, situations fhat have to do with one*s occupation, 
with dilienshlp and aalntenace (private life), with a ftm adapUtlons to 
the Ihitch situation, the list proved very helpful. 
^^^!^.$^^' contents iia,*e Urgaly detenrtned by the relevance of 
VilVAV " ^mrmt ffm thoir ftetuHne 1p toxthoohs. ^ 
suDSt¥iie ar^t 
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rtformjliting, fMsoolnc. distinguishing wjor mtUr% fnm «lnor 
onts. ranking facts chronologically* oxprtssing an opinion and 
(for tho catogoflos othar thm tha mDloMa) asMno and answering 

rU!^* ?*M^^»9»^P^'^<«9 to^Sa fxprasslon of actions. 
^.1^^^ •^^^ ^ assoMMnts to bt iada, ms basad on 
tha folloiHnf contldaratlon. Sinca tht ttsts am mot for classroon 
usa anahling toachors to ascaruin tditlhor or mt thoir pupils comnd 
a particular coiiponont of ^ral proflclancy afttr Instructions or 
without spaclal training as tho caso aiylo-anBlytlc assassaant 
on tha basis of a nuibar of crItaHa irtll ba aon In tha Interest 
of tha pupil and tho toachar thin global assoaiMt. In this case 
agraaaant awmg ratort vould soaa to bo lau l^orUnt than 
conslstancy of tha ratings ovor a paHod of tiaa. 

JiL! ^"^^ co*Mnod Into tasts. 

- spMck tituatlM {mabar of Mtatort) 

- SUikjKt ■•tt«r/C(Mt«Xt 

- tubtkll). 

^ h LKL-Ili' • "HWU fill about 

10 aio niB m aon got o 'iit of ■ aiatar of okllMtary points. 
Such • mt provod M otsMtlol altf tOMMyX • TiirMiv 
221 !: ms to bt fUnr bImCw Io^. U U cle.r 

MtMt Mt art »fttr 1« tklt itslgMMtt tht Mil ilM to ^vo a report 
1; I" XeH Jght occur 

SL'2!S2ii? i; Ji'y?"*.^ it< o ^rtm.W Kt Dm to dSonstraU 
tho MbiMII of 41tt1o»i1«h1«fl Mmm aUtr kK aiiBr atttors. 



^i,K*J^ C^ltlo^it') toit puts tkt pHtll la a situation In 
j*lc* h« hatto tjrry on a sotivorsatloo. net tht InfUrMl iveryday chat, 
but rathor a aoro or loss fbiwl InttrvlaM. Tkt iubII Ii auiactad 
to acjuirt infoiwtlon by oslxlnf ^,JSmZ inim,^^ • 
tItuatfoM. Ftr iMUnco. he Is siMOtti tt bt MtntlM Inforaatlor. 

iHt S i}J5*'*'?^' JtlMtg. abOMt • ftr^piltrtrganliod trip 
that ho Might Mnt to atttr for. Tkt p«11 flmW a s!»tch of tho 
sltuatloo with tho otslimnt to *sl.7lth1~f?55 iirtt spal.^j 
2l-S!;Jir!<2T»!'* 2 «*ta1n,tht r««i1rti It1^?i5!;. 

fe^^^lSf JJi tt i^igtlH^nfOnMtlon a$k«4 
«ri.J^k?^.w'.^* V**' •? • •«»1««» •^Ich tho teacbor 

ttloctt h1»*1f ho aim only tvulvo answars rtwirino the pupil to 
kaop on asking quosilOM uttll ho hM rtct1»t< snrtc oar 

ansMT. Tht pupil mmt mt lot hiMtIf bt ptk 

^ l^t^ «' » third tost, a 'polyloiM*. COMS undtr 

■tant for IS to 14 ytar olts. Such a Mbjtct Is^publlc transport'. 

erJc b^t copy available 
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«i«l2'fJ2!'^i' JJjI'.i!**.?"*!* to laltiu as lirg* ■ 

Sblir^ »n«tions In Mhlch onl spMch is umT m 

3 hnt- tMtt end rating Mtslont 

1— nir/'^Jl!* «»»J»*1nts Involvwt In the Ust tItMtlon U 
iT^IlSJl**" "^""^ «>^»1our that iTtlkl^ can 

bt isMtttd proptrly. To uttmr that qutstlon prattttt aid rltlna 

|y WMs of tkt ^tttts M hoiMd to find out Si bSt" 

roillW1R9 Ctttt Hirt WMd: the WMOlMlM<-tMt ttenrt am b PI1«. 

fo«?i2,s,':},re "KSd*i£? *^ of t*» 

MsalOM tMrt sM up liko this. 
KPOftno MniB S(S$IONS 

MTED OITEIIA SCALE fiLSM. 
CMB. MINTS MTIW 



No. 


NuwaoF 

MTEIS 


TEST 


1. 




DIALOfiUE* 
MBMEXINB 


2. 




OIALOOJC* 

MSMaw 
qucsT;o|B 


3. 


12 


NONOIOGUE 
tf FILM 



•lOCNTICM. 
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•n txUntlj; Sdll SitSi^i^^^ f^Umrt used: 
«cilt it cht firtt So t^rfJ. cHttrfi n four-point 
CPittPii and • d1ff!S«?^S?!:iL*'*,^** •^th four 



WTIMB NOKL SESSION 1 ind 2 OlilLCfiUE 
COHTDIT 

1. brinjs up th« otilgitory points 

2. bpi^9j up points of Ms own 
clioos1n9 



3. asks for ntotssary o)tfc1<tetion 

4. rtpooti Hlmolf ufiMc»ss#r11y 

5. words Md sontonpos aro su1t«bU 

6. pronunciation asid wrds arv 
non-standard ^ 

OCIIVERY 

7. sptaki cloarly 

8. sptaks fluantly 

9- sptaks Bonotonously 

cmiimcATiai 

W. Minuins contaa with 
iitATlocutor 

11. InttmipU tht Inttrlocutor 
GlOaAL MTIN6 (Mrlts froa l-lO) 



SCALE WIHTS 
YES m 

0 

a 

H!L •f^^ 

TWES ALWAYS 

• P 0 0 

• 0 0 0 

0 

0 

0 
0 
0 

0 

0 



0 
0 
0 

0 
0 



0 
0 

0 
0 
0 
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RATING NOOCl SESSION 3 NONOLOOIE 



CRITERIA 



SCALE poiprrs 

WEAK BARELY FAIR EXCELLEITT 
SUFFICIEKT 



0 
0 
0 
0 



1. CONTENT 0 0 0 

2. (MAKISATION 0 Q o 

3. LANGUAfiE 0 0 0 

4. oaivERy 0 0 0 

aORAL RATINfi 

3 1 Ccnslittm;/ and rtl lability of raUrs 

Vm first qutstlofi I Mntloiiad abovt coficamad tht rtllablllty 
and aHit1staiM;y of tlit raUrs. Conilittncy Is, of courtt, an 
asptct of ^liability. Tht conslsttncy jf rattrs could b4 

MMid bt 

ylll not bt ablo to onHst tbt btip iF ath^ i^ttrtr'lMs'ldMww 
Sl^r^flL!!?'' ^ ^ coi»1st«»t la tlitir assasMts tliui 
Jortjacjorj ieo agrot wi^Ji ascb otber: coaslstaiicy Is mr% 
HiportMt tim Inttr^nttr aa 




TAH.I I. COEFFICIENTS OF CONSISTENCY IN DIFFERENT RATIN6 SESSIONS 



RATER I 


.86 


RATER Z 


.05 


RATER 3 


.79 


RATER 4 


.04 


RATBI S 


.73 


RATER a 


.73 


RATIR 7 


.00 


RATER 6 


.73 


MTCR 9 


.79 


RATER 10 


.74 


RATER n 


.86 
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Thtst corrtlitlons art rtlat^vtly Nigh Mlilch Mans that the raters 
are fairly comltUnt In thtir judgcMnta. 

In order to ietervlne the rater's rtllablllty - in the sense 
of thilr hoaooenelty - a iMogenelty analysis was used. This 
procedure aislma mabers to scelepolnts for each rater. These 
nuiibers are called scalevelues. |y Mane of thtst scalevalues the 
original date (the feting theets filled out by the reters) can be 
translated Inte a mmrlcal daUtahle. For each retor the 
corrtlatloni hetMoen the oeroelved rating and the seen (« true) 
ratine can he couputed. The sqyan of that correlation u the 
rater U rellebf^lty. 

Hoaogenelty a»>a1ys1s detenalnes the scalevalufe for the scalepolnts 
n such a nay. 'Jiat the wan rater reliability - or hoaogenelty - 
Is as high as possible.) 

The rater's reliabilities for each of Vn three sessions were as 
follows. 



TABLE 2. MTER RCLIMILJTIES-ITISCftmillAnOII MEASURES 



RATER 


RATING 
SESSION 
1 2 


RATER 


RATIIC 
SESSION 
3 


1 


.80 


.81 


1 


.43 


2 


.74 


.74 


2 


.67 


3 


.62 


.69 


3 


.38 


4 


.71 


.73 


4 


.58 


5 


.5i 


.59 


S 


41 


6 


.SO 


.72 


6 


.43 


7 


.68 


.67 


7 


.55 


8 


.63 


.64 


8 


.43 


9 


.84 


.76 


9 


.61 


10 


.63 


.S9 


10 


.51 


11 


.73 


.75 


11 
12 


.70 
.62 


Mean 


.68 


.70 




.55 



Th« dIfftrtKt In mm rtllablllty bttWM Uw first tM 
mtlont am tiM tlilr^ un ptriMpt b« Kcouit.4 far tkt fact 
that 41fftrMt ratln* f«»Mtt Mf« uit4i tiM flrit f»nMt utti 
tItVM crittria and foMr tcaltpolnts. tkt ««okJ fMWt wad fo>ir 
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crlttrU (ini Win but dlfftnwt scilw^InU). TM four 
crittrii of thi Mcond fotmt aty bo too conplex to l« wHo^ 
UMSbl Quoutly « 

High MOfi rolltbnity dots not mm of courtt, .Hot ill rotor* 
•gniod In tbolr issttMiits. Tho mlytls only ittonrlnot tlio 
ritort* tangtnolty. thot It. tho oxtofit to irfilo, tDoIr ismaMiits 
pilr, or ilso tlio oitoRt to ulildi ono rotor's ismtMRt con bt 
prodlctod 00 tHo boon of onotlior rotor's issosoM^t. Thm ritart 
will only givo aoro or loss slallir issoststnU, If tlio scilo 
viluos tkit «ro usIfMd to tbo poinU fur ooe^ rotor rosoH^lo 
oach otbor por P*<«t. tbit Is, If thty Mvo only llttlo 
virlirco. THo scolo viluos obUlnod In th« thrto sotslono con bo 

SHOHSOd 00 fOllOM. 

TMLC 3. SCALE VALUES. MEANS AND STANOAROOEVIATIONS. 



RATING 
SESSION 


SCALEKINT 


MEAN SCALE 

MAI iir 


STAKMRD 


1 


lARELV 




• V/ 




SOMHMES 


.00 


.19 




MTEN 


.96 


.19 


2 ^ 


AUOST ALWAYS 
NAKLV 


\,IZ 
'M 


.16 




SONETIKES 


.11 


.18 




OFTEN 


•7 


.16 


3 


ALNOST ALIiAYS 
UEAK 


l.M 
-1,21 


.19 
.22 




lAHaV SUFFI. 


-.13 


.14 




FAIN 


M 


.11 




EXCELLENT 


1.3t 


.26 



Frw tlM dlff^roncos botMOon tho noon valuos It can bo 
concludod tbot tbo rotoro fwntly usod tho •Ppjjj** 
Mhlch MOM thot tlMty igrood in thoir issMMonu. Forif this Moro 
Mt tho coto. If thoy hod dlfforod aoro or lost rindMly tiM and 
ap1n» tlM Mon scolovalyM would hovo bMn Idontlcol. 



frm tho fact that. In all thrM sotslons, thoro Is a rvgular 
Incrooio of tno MM vilwo (ovm though tho scalo ^Ints usod aro 
not tN soM) It Is ciMr that tio ocalo polntt havo boon Intomtod 
sMlarly to « corUin MUnt, whloU U U s^ thot^ f^r oxa«p1o» 
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ofttn* Ms not syst«ut1ca11y bmn rM«rxtod is iuptrlor to 
'soMtlMS* In ciSt of t particuli^- cHttriow by um rittrs. 
***!!",?^r.!:^^*'* ^•^^ P«»^«t ttm othur vcy 

^ "Mn viliMS clMPly show wh«fl 
mlltblllty Inttrvils art conputtd on tht basis of tho sUndard 
aovlatlons found. As far as the coIumi of standard deviation Is 
conctmwl: the mller the SD of the scaleveluet assigned to a 
scalepoint Is, the aore frt^imtly this scelepolnt has been used 
by all nters at the saM ttM. These sUnderd deviations are lour. 
Those for 'weak* and 'excellent* are hlohest fdr the third sess.on. 
This Is In accordance with the relatively low ■neci rater rellatiUty 
for this session, as was shown In TABLE 2: the^e two ftcalepo1n:$ 
were apparently InUrnreted differwitly by dlffereiit nUn. 

3.2 Independence of cHterIa 

The second question regarded the Independence of criteria. It 
was expected that the criteria coutd not bTapplled entirely 
Independently, since they relate to one conplex skill, but on the 
other hand th^ were not supposed to be CMmected too closely, as 

llLi^^!i!f ^??fi; ^^^\^\^ ^^'^ ^ crtterlon wouid suffice. 
And then the so-called analytic assesiMnt would not :iave any 
advantages over a global one. Analytic essetiwit has the advantage, 

lI15«'iJ!l[2'^ ?I ^ teacher cin 

^."Xi^^^*,*^ ^^^^ W^^' »k111s fall short 
and call for additional training. 

The howgenelty of the cr;ter1a has bnen da.^nilKMi by aeans 
of the SIM scale analysis wcs was used for conputfng reter 
reliability. For each of the three sessions a neth cHUrlon was 
deurwined. If all crIUrIa correlated ht#i with thH Man criterion, 
In fact one criterion would suffice. The next table shows the squa re 
correlations bettwen the cHUrle end the mn cr«,^on. 



BfcST COPY AVAILABLE 

ERJC lU 



-146" 



TABIE 4, 


DISCRIMINATION MEASURES CRITERIA 


CRITERIA 


RATING SESSION 


CRITERIA RATING 








SESSION 




1 


0 

•> 


3 


1 




. w 


I .62 


Z 




.Ui 


2 .72 


3 




IK 


3 .59 


4 


AH 


.u/ 


4 .66 


S 


• DO 




MEAN .65 


6 




• ^o 




7 


Al 


.67 




8 


.S7 


.SS 




9 


.40 


.57 




10 


.57 


.AO 




11 


.02 


.00 




NCAN 


.33 


,33 





Tht Mill corrtUtlons ir« 1w. partlculiry for tht first tMO 
stsslons. Dilt mum dut tlit crltorli do not ovtrlip coopltttly. 
This Is MlRly duo to tNt critoria In tht first cittgory, that 
of CONTOIT. Togtthtr with crlttrlon no. U - Mntomipts 
Intorlocutor* - thty art dorlint. Th« four crittria ustd in tht 
third stsslon, which art u a aatttr of fact tht htadlngs of tht 
four catt^Hts In tht first fomat, all corrtlatt «ort clostly 
with tht atan crlttrlon. (Thtst corrtlations cannot bt sinply 
ccsir«rtd to tht othtrst btcaust tht cHttHa as Mil as tht scalt 
points itfm^ frm thost Mtd In tht first tM stsslons.) 

In ordtr to tstabllsh to Mhat txttnt tht cattgorlts thtMtlvts 
art hoMjUMom, tht corrtlations bttnttn tht various criUria and 
tht Mtn cHttHon ptr cattfory hawt Ntn coi^uttd u Mil. 
Iht ntst tahlt shtm a SMrvty of tht (s^rt) otrrtlatloni. 

i 
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TA8LE S. DISCRfMIHATIOW NEASU«£S PER CAfEeORY Of CRITERIA 









SESSION 2 


COMTENT 


1 


.32 






2 


.26 


.06 




3 


.47 


.00 




4 


.24 


Kft 


LANGUAGE 


K 

if 


.78 






r 


.78 


. '3 


OaiVERY 




.71 


.7^ 




8 


.SiR 


.69 




9 


*S8 


.70 


OmMICATION 


HO 


• 66 


•57 




11 


.t6 


.17 



TABLE fi. CORRELATIOHS BETMEEH CATEGORIES 
WTIMG SESSION 1 r^tIHG SESSION I 

CitZ .25 Cita .M 

Cit J .27 .60 c,t 3 ,n M 

ct 4 . 05 -.01 .06 4 .J3 .04 -.19 

Cit 1 Cit 2 Cit 3 ctt 1 «t t Cit J 

^ "i^ry 'tould ot split up; 

BEST COPY AVAILABLE 
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reJucing the numb.r of criteria to no more than the category 

the categories of LANGUAGE and OELIVtPY in the first fonnat are 
the ones that could be combined ^ 
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Analytic and global rating 



The last questio,. that we asked ourselves regarded the 
0 ec^e'd da^' d f assessn^nt: global and anaiy k. The 

collec.ed da.d do no^ point to a def.nUive conclusion Wp 

o^t b"l^ LuUs^'h^ assessment and this preference' Is'not ruled 
uuL ov ^ne results. Here are some fi»^'jres 

' Z "iMons's 'eT" ^-^^ 

^ [h! °" the mean criterion and 

the global marks given by the raters are 
session 1; 78 
session ^. .55 
session 3: 89 

3 The first principal components of session 1 and session 2 fas 
oT Tl ' hu" f e^f tT"' " "■'^^'■"^'^^ show"'correli?ion 

assessment "'•f'^l ation between the two methods of 

" r«snhrra?ers" ^ ^"^^"^ °^ -^P^^^ - 

- for classroom use, when only one teacher ran assess the 
acnievenents. the analytic assessment ,s to be preferred. 



^ Sibl lography 

Sassett, R.E. et a,. '.78. 'Basics m speaking and listemnq' 

Commjnjicat20n_^ 1978: 293 f_ ' nsiemng . 

'"'\'n$°i'4- nnf - of categorical data, dual scaling 

and 1 vS appl ications. ^ 

Toronto University of Toronto Press 

"'"""?::^da"sCo/'''- discussievaardigheid. 
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